Collecting Data for Comparability: Benchmarking Software Development Productivity

0 7 4 0 7 4 5 9 / 0 1 / $ 1 0 . 0 0 © 2 0 0 1 I E E E Planning for data collection You can’t benchmark data if you haven’t collected it. Writing a questionnaire and having people complete it is not enough—you need a vision. It’s similar to getting the requirement specifications right before developing software: if you learn that something is wrong with the data after collecting and analyzing it, your conclusions are meaningless, and you have to redo your work. I have wasted months trying to make sense of data collected without a clear purpose and without statistical analysis requirements in mind. If you work for a large company, consider asking the market or operations research department to help design your benchmarking questionnaire. Software managers know about software; data analysts know about questionnaire development. Collecting the right data for your purposes might require a multifunctional team effort. Regardless of whether the data concerns chocolate bar sales, financial indicators, or software projects, the old maxim “garbage in equals garbage out” applies uniformly. Make sure that the variable definitions and responses are clear before you collect the data. Typical questions I ask when validating software project databases include: What does a zero mean? Does it mean none, is it a missing value, or was a number close to zero rounded to zero? And if a value is missing, does that indicate no value, don’t know, or, if the question involved choosing from a list, was the correct response missing? Lists that include Other as a choice are also problematic, especially when collecting data for benchmarking. For example, let’s assume that your company questionnaire includes a list of case tools. The case tool used on your project does not appear in the list, so you select Other. This focus